Sparse Linear Programming via Primal and Dual Augmented Coordinate Descent

Over the past decades, Linear Programming (LP) has been widely used in differentareas and considered as one of the mature technologies in numerical optimization.However, the complexity offered by state-of-the-art algorithms (i.e. interior-pointmethod and primal, dual simplex methods) is still unsatisfactory for problems inmachine learning with huge number of variables and constraints. In this paper,we investigate a general LP algorithm based on the combination of AugmentedLagrangian and Coordinate Descent (AL-CD), giving an iteration complexity ofO((log(1/ ))) with O(nnz(A)) cost per iteration, where nnz(A) is the numberof non-zeros in the m×n constraint matrix A, and in practice, one can further re-duce cost per iteration to the order of non-zeros in columns (rows) correspondingto the active primal (dual) variables through an active-set strategy. The algorithmthus yields a tractable alternative to standard LP methods for large-scale problemsof sparse solutions and nnz(A) mn. We conduct experiments on large-scaleLP instances from `1-regularized multi-class SVM, Sparse Inverse Covariance Es-timation, and Nonnegative Matrix Factorization, where the proposed approachfinds solutions of 10−3 precision orders of magnitude faster than state-of-the-artimplementations of interior-point and simplex methods.